Harden hypervisor process liveness checks - #363

resolveRuntimeHypervisorPID minted a boot-scoped identity token for a
fallback PID that ProcessExists had just disproven, stamping the current
boot ID (and, if the PID were recycled mid-call, a live start time) onto
a process that is not the hypervisor. Record the bare PID with a cleared
token instead, matching the command-line-only branch.

shutdownHypervisor SIGKILLed the raw stored PID and removed the control
socket without the ownership checks the other kill paths enforce, so a
recycled PID could be signaled and the fail-closed stop fallback was
undermined. Resolve the live owner first: kill the resolved PID only,
skip the kill when the recorded hypervisor is provably gone, and fail
closed before any teardown when ownership cannot be confirmed, keeping
the socket as evidence for the hardened kill path.

When the control client cannot be built but the resolved socket owner is
alive, shutdownHypervisor force kills the owner instead of reporting a
completed shutdown and unlinking the socket over a running VMM.

A command-line-only match that is already dead now classifies as
provable death rather than failing closed: the socket-owner scan found
nothing and no live process matches the command line, the same
conclusion as ErrNoOwningProcess. The classification moves into
classifyResolvedHypervisorOwner so that race is directly testable.

refreshHypervisorPID ran resolveLiveHypervisorPID on every hydration of
every running instance, so each list/get paid socket resolution — worst
case a full /proc scan — for instances mid-transition. Hydration is
read-only and its answer never authorizes teardown: stop, delete,
standby, and the vGPU release guards all re-resolve identity before
acting. Restore the cheap contract: trust a live stored PID, resolve
the socket only when it is dead, and mint the identity token only for
a confirmed owner. Durable token backfill for legacy metadata stays
with BackfillHypervisorProcessIdentities at startup, which persists it;
hydration's in-memory upgrade never did.

Embed HypervisorProcessIdentity in StoredMetadata anonymously so the
persisted JSON keys are unchanged, replace the nine hand-cleared
(PID, StartTime, BootID) sites with Set/SetUnconfirmed/Clear methods,
and pass the struct to resolveLiveHypervisorPID instead of three
positional fields.

killProcessAndWait replaces the kill blocks in killHypervisor,
forceKillHypervisorProcess, and forceKillHypervisorPID. All callers now
get the process-group escalation standby already had, and the SIGKILL
wait used by stop and delete is a named constant.

Count backfilled, skipped, and failed instances and emit one info line
so a wholesale backfill failure is visible without debug logging.

A hypervisor that survives SIGKILL is stuck in uninterruptible sleep and
no amount of waiting unsticks it, so the long wait only slowed down stop
and delete. killProcessAndWait still escalates to the process group and
grants a 2s grace period after the initial wait.

Delete no longer fails when the hypervisor cannot be confirmed dead or
instance data removal fails. Instead the instance is marked
pending-delete: it disappears from list, get, and name lookups (freeing
its name for reuse), the delete lifecycle event fires, and the API call
returns success. A background finalizer retries the remaining teardown
until the hypervisor is provably gone, including a pass at startup where
the boot-scoped process identity settles the stuck case immediately
after a host reboot.

Admission capacity stays held until finalization since the stuck
hypervisor may still pin its memory and devices, and TAP GC preserves
pending-delete TAPs for the same reason. A new
hypeman_instances_pending_delete_total gauge exposes wedged teardowns.

This reverts commit 2559a7d.

HypervisorProcessIdentityExists and HypervisorProcessExists have no callers
in this change; the vGPU work that consumes them introduces them itself.
killProcessAndWait always waited 2s regardless of which of the two
identically-valued constants callers passed, so the parameter goes away.

The resolver disproof branches (boot mismatch, start-time mismatch) were
each tested through killHypervisor with subprocess scaffolding; test them
at the resolver level instead and keep one wrapper-level no-op test.
Fold the classify fail-closed/provable-death cases into one table test
and drop the kill-level duplicate. Replace the three backfill tests that
only exercised the needsHypervisorIdentityBackfill predicate with a
direct table test.

Both resolved the live owner and called killProcessAndWait; the only
differences were an early return already handled inside
resolveLiveHypervisorPID and a trailing socket remove that is a no-op on
stop's path (shutdownHypervisor already removed it).

standbyInstance logged and continued when shutdownHypervisor failed,
then released the TAP and cleared the process identity — written when
that error meant only "graceful API failed after we killed the raw
PID". With fail-closed ownership checks the error can now mean nothing
was killed, so continuing would release the network under a live paused
VMM and erase the only identity that could ever find it again.

Standby now resumes the VM and returns the error; the snapshot on disk
is harmless and a retried standby redoes it. shutdownHypervisor's
trailing graceful-API error is demoted to a warning since it only
fires after the process is confirmed gone, giving the function a
single contract: error means the hypervisor may still be running.

shutdownHypervisor unlinked the control socket via a defer on every
return and again before waiting for the process to exit. On the paths
where the kill fails and standby resumes the VM, that left a live
hypervisor with no socket file, so a later graceful standby or stop
could not connect and fell straight into the force-kill path. Unlink
the socket only once the hypervisor is provably gone, matching the
contract killHypervisor already follows on delete and stop.

A process matching the socket path by command line only can never
authorize teardown, so the fallback's sole effect was distinguishing
provable death from ambiguity. The fd scan runs with CAP_SYS_PTRACE
(hypeman runs as root or with full caps), so it cannot miss a live
owner and a missing listener already proves the hypervisor is gone.
The fallback was also actively harmful: a debug client holding the
socket path in its argv (ch-remote, socat) resolved as an unconfirmed
live match and wedged stop/delete until it exited.

Resolution now trusts the listener scan alone: a confirmed owner is
returned, no owner classifies as provable death, and only a failed
scan fails closed.

shutdownHypervisor returns nil only when the hypervisor is confirmed
gone, so the unconditional killHypervisor that followed was a redundant
re-resolution. Run it only when shutdown fails.

Also make pidBySocketRef prefer the expected owner when the full scan
finds the listener fd in multiple processes: the scan's observation is
the same evidence the fast path uses, so a child transiently sharing
the inherited fd must not turn a proven owner into a fail-closed error
when the fast path misses on a transient fd-dir read failure.

The backfill existed to persist identity tokens for legacy metadata so
destructive paths could skip socket resolution. Correctness never
depended on it: a token-less live PID is confirmed through the
expected-owner fast path (one /proc/net/unix read plus one fd-dir read),
and after a reboot the empty listener scan classifies the recorded
hypervisor as provably dead. Stop, delete, and standby re-resolve
ownership on every call regardless, and instances mint tokens on
create/restore, so the startup pass and its metadata rewrites bought
nothing.

resolveLiveHypervisorPID treats a zombie stored PID as dead, so the
confirmed-gone paths in stop, delete, and standby never reach the Wait4
in WaitForProcessExit and leaked one zombie per direct-child VMM that
exited on its own. Reap it non-blockingly where the dead PID is observed.

TestResolveProcessPIDDuringProcessChurn passed the expected owner as a
hint, so the fast path answered from the test process's own fd table and
the full /proc scan the test exists to exercise never ran. Resolve
without the hint.